SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Australian Web Archive</title>

</head>

Australian Web Archive

</h1>

</header>

The Australian Web Archive (AWA) is a publicly available <a href="Online_database" class="mw-redirect" title="Online database">online database</a> of archived Australian websites, hosted by the <a href="National_Library_of_Australia" title="National Library of Australia">National Library of Australia</a> (NLA) on its <a href="Trove" title="Trove">Trove</a> platform, an online library database aggregator. It comprises the NLA's own <a href="PANDORA_archive" class="mw-redirect" title="PANDORA archive">PANDORA archive</a>, the <a href="Australian_Government_Web_Archive" class="mw-redirect" title="Australian Government Web Archive">Australian Government Web Archive</a> (AGWA) and the <a href="National_Library_of_Australia" title="National Library of Australia">National Library of Australia</a>'s ".au" <a href="Web_domain" class="mw-redirect" title="Web domain">domain</a> collections. Access is through a single interface in Trove, which is publicly available.<a href="#cite_note-1">[1]</a><a href="#cite_note-2">[2]</a><a href="#cite_note-3">[3]</a> The Australian Web Archive was created in March 2019,<a href="#cite_note-bruns-4">[4]</a> and is one of the biggest <a href="Web_archive" class="mw-redirect" title="Web archive">web archives</a> in the world.<a href="#cite_note-nott-5">[5]</a> Its purpose is to provide a resource for historians and researchers, now and into the future.<a href="#cite_note-nott-5">[5]</a>

<div class="mw-heading mw-heading2"><h2 id="History_of_the_three_components">History of the three components</h2></div>

The PANDORA service started archiving websites in October 1996.<a href="#cite_note-hist-6">[6]</a>

In 2005, the NLA started archiving annual snapshots of the entire Australian web domain (<a href="URL" title="URL">URLs</a> with the <a href="Suffix" title="Suffix">suffix</a>. ".au"<a href="#cite_note-bruns-4">[4]</a>),<a href="#cite_note-preserve-7">[7]</a> collected via large <a href="Web_crawler" title="Web crawler">crawl harvests</a>.<a href="#cite_note-awa-8">[8]</a> Later, the earliest websites from the .au web domain, dating back to 1996, were obtained from the <a href="Internet_Archive" title="Internet Archive">Internet Archive</a>. In 2019 this content was first made publicly accessible through Trove.<a href="#cite_note-helpabout-9">[9]</a>

The PANDORA infrastructure, which works well for a selective small scale archiving, does not adapt to large scale "bulk harvesting" of web content, so a new technical system had to be developed whereby a web archiving service which would integrate the delivery of archived websites within a live website interface delivering the archived websites seamlessly to the user, which is difficult to achieve technically.<a href="#cite_note-koerbin2015-10">[10]</a>

<a href="Australian_Government" title="Australian Government">Australian Government</a> websites are Commonwealth records, and are therefore publications to be managed in accordance with the Archives Act 1983.<a href="#cite_note-naa-11">[11]</a>

The Australian Government Web Archive (AGWA) consists of bulk archiving of <a href="Commonwealth_Government" class="mw-redirect" title="Commonwealth Government">Commonwealth Government</a> websites. The NLA began regular harvests of the websites in June 2011,<a href="#cite_note-12">[12]</a> after a significant obstacle had been overcome with an administrative agreement made in May 2010 allowing the NLA to collect, preserve and make accessible government websites without having to seek prior permission for each website or document, as was the case before that. The service uses the <a href="Heritrix" title="Heritrix">Heritrix</a> web crawler for harvesting, <a href="WARC_file" class="mw-redirect" title="WARC file">WARC files</a> for storage and Open Wayback for delivery of the service. There is a huge amount of publishing by the government, but many challenges to overcome trying to preserve content, such as its sudden disappearance. In March 2014, the AGWA was made publicly accessible.<a href="#cite_note-koerbin2015-10">[10]</a>

The AGWA meets the preservation and retention requirements for websites as "retain as national archives" (RNA) material under the Archives Act; however <a href="Video" title="Video">videos</a> and document files ( such as <a href="PDF" title="PDF">PDFs</a> or <a href="Word_document" class="mw-redirect" title="Word document">Word documents</a>) are not always captured, so must be managed separately.<a href="#cite_note-naa-11">[11]</a>

As of early 2015, the AGWA included content dating from 2005, which amounted to about 144 million files occupying 15 <a href="Terabyte" class="mw-redirect" title="Terabyte">terabytes</a>. It only included Commonwealth Government websites collected through bulk harvests of nearly 1000 seed URLs. The scheduling of the harvests was not yet routinely established, but harvests were being conducted roughly three times per year.<a href="#cite_note-koerbin2015-10">[10]</a>

<div class="mw-heading mw-heading3"><h3 id="Amalgamation">Amalgamation</h3></div>

In 2017, the AGWA and the PANDORA archive were amalgamated with the other web archive collections, to form the Trove web archive collection.<a href="#cite_note-helpabout-9">[9]</a> After further development and the creation of the Australia Web Archive, government websites archived via AGWA and now included in AWA can still be searched separately using the "Advanced Search" option.<a href="#cite_note-helpabout-9">[9]</a>

<div class="mw-heading mw-heading2"><h2 id="Description_of_AWA">Description of AWA</h2></div>

A web archive is described by the NLA as a "collection of snapshots of websites captured while they are accessible on the web, and then preserved in a static copy". The collection archived in the AWA is "relevant to the cultural, social, political, research and commercial life and activities of Australia and Australians". It collects web material via both scheduled archiving of selected websites and publications as well as some <a href="Ad_hoc" title="Ad hoc">ad hoc</a> harvesting relating to significant events.<a href="#cite_note-helpabout-9">[9]</a>

As of March 2019, when it began, AWA already contained around 600 <a href="Terabyte" class="mw-redirect" title="Terabyte">terabytes</a> of data, with 9 billion records.<a href="#cite_note-nott-5">[5]</a><a href="#cite_note-13">[13]</a> It contains more functionality than the <a href="Wayback_Machine" title="Wayback Machine">Wayback Machine</a>, hosted by the <a href="Internet_Archive" title="Internet Archive">Internet Archive</a>, allowing <a href="Full-text_search" title="Full-text search">full-text searching</a> using a <a href="Search_engine" title="Search engine">search engine</a> built in-house. The developers also devised techniques to filter out unwanted "noise". The data remains on the Library servers, although a move to the <a href="Cloud_storage" title="Cloud storage">cloud</a> is envisaged in the future, as content grows.<a href="#cite_note-nott-5">[5]</a> Usability by a wide range of users, and in particular the search functionality, were major focuses during development.<a href="#cite_note-helpabout-9">[9]</a>

The archive is fully searchable, based on a combination of techniques used by the developers. Each team created a unique and complex <a href="Search_algorithm" title="Search algorithm">search algorithm</a>, by adapting a version of <a href="Google" title="Google">Google</a>’s page ranking algorithm (based frequency of clicks on a page), modified to lead to better, high-quality resources. Other technologies include a <a href="Bayesian_filter" class="mw-redirect" title="Bayesian filter">Bayesian filter</a> (effectively a <a href="Spam_filter" class="mw-redirect" title="Spam filter">spam filter</a>), a <a href="Not_Safe_For_Work" class="mw-redirect" title="Not Safe For Work">Not Safe For Work</a> classifier from <a href="Yahoo" title="Yahoo">Yahoo</a>, and <a href="Machine_learning" title="Machine learning">machine learning</a>.<a href="#cite_note-phone-14">[14]</a>

There is a "Limit to the gov.au web domain" option before searching,<a href="#cite_note-hometrove-15">[15]</a> and government websites archived via AGWA can still be searched separately using the "Advanced Search" option.<a href="#cite_note-helpabout-9">[9]</a> Other options in Advanced Search are to limit by timespan of the snapshots, domain and file type.<a href="#cite_note-16">[16]</a>

With many of the earlier websites from the 1990s now lost, mainly because of the frequent change of web platforms, the Australian Web Archive is a significant initiative that will help to save current and future web pages, especially Australian content.<a href="#cite_note-bruns-4">[4]</a> Material will continue to be added to the Archive, and other online material collected in accordance with the National Library Act 1960, the <a href="Legal_deposit" title="Legal deposit">legal deposit</a> provisions of the <a href="Copyright_Act_1968" class="mw-redirect" title="Copyright Act 1968">Copyright Act 1968</a> and the NLA's <a href="Digital_collections_selection_criteria" title="Digital collections selection criteria">digital collections selection policy</a>.<a href="#cite_note-helpabout-9">[9]</a>

<div class="mw-heading mw-heading2"><h2 id="Asia/Pacific_websites">Asia/Pacific websites</h2></div>

Websites in the <a href="Asia_Pacific_region" class="mw-redirect" title="Asia Pacific region">Asia Pacific region</a> are not included in the AWA, but NLA partners with the <a href="Internet_Archive" title="Internet Archive">Internet Archive</a> to collect and preserve "selected Asia/Pacific websites related to specific events or socio-political groups".<a href="#cite_note-National_Library_of_Australia_2020-17">[17]</a>

<ul><li><a href="National_edeposit" title="National edeposit">National edeposit</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://pandora.nla.gov.au/">"Preserving and Accessing Networked DOcumentary Resources of Australia"</a>. Pandora Archive. Retrieved 30 April 2020.</cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.nla.gov.au/what-we-collect/archived-websites">"Archived websites"</a>. National Library of Australia. 23 March 2020. Retrieved 30 April 2020.</cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFKoerbin2015" class="citation web cs1">Koerbin, Paul (11 February 2015). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20200430104945/https://www.nla.gov.au/stories/blog/web-archiving/2015/02/11/the-australian-government-web-archive">"The Australian Government Web Archive"</a>. National Library of Australia. Archived from <a rel="nofollow" class="external text" href="https://www.nla.gov.au/stories/blog/web-archiving/2015/02/11/the-australian-government-web-archive">the original</a> on 30 April 2020. Retrieved 30 April 2020.</cite>

</li>

<li id="cite_note-bruns-4">^ <a href="#cite_ref-bruns_4-0">a</a> <a href="#cite_ref-bruns_4-1">b</a> <a href="#cite_ref-bruns_4-2">c</a> <cite id="CITEREFBruns2019" class="citation web cs1">Bruns, Axel (14 March 2019). <a rel="nofollow" class="external text" href="https://theconversation.com/the-australian-web-archive-is-a-momentous-achievement-but-things-will-get-harder-from-here-113542">"The Australian Web Archive is a momentous achievement – but things will get harder from here"</a>. The Conversation. Retrieved 30 April 2020.</cite>

</li>

<li id="cite_note-nott-5">^ <a href="#cite_ref-nott_5-0">a</a> <a href="#cite_ref-nott_5-1">b</a> <a href="#cite_ref-nott_5-2">c</a> <a href="#cite_ref-nott_5-3">d</a> <cite id="CITEREFNott2019" class="citation web cs1">Nott, George (11 March 2019). <a rel="nofollow" class="external text" href="https://www.computerworld.com/article/3488134/national-library-launches-enormous-archive-of-australia-s-internet.html">"National Library launches 'enormous' archive of Australia's Internet"</a>. Computerworld. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-hist-6"><a href="#cite_ref-hist_6-0">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="http://pandora.nla.gov.au/historyachievements.html">"History and Achievements"</a>. PANDORA. 18 February 2009. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-preserve-7"><a href="#cite_ref-preserve_7-0">^</a> <cite id="CITEREFMcKenzie2019" class="citation web cs1">McKenzie, Amelia (12 March 2019). <a rel="nofollow" class="external text" href="https://www.nla.gov.au/stories/blog/trove/2019/03/12/preserving-australias-web-history">"Preserving Australia's Web History:The beginning of the Australian Web Archive"</a>. National Library of Australia. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-awa-8"><a href="#cite_ref-awa_8-0">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://trove.nla.gov.au/website">"Archived websites (1996 – now)"</a>. Trove. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-helpabout-9">^ <a href="#cite_ref-helpabout_9-0">a</a> <a href="#cite_ref-helpabout_9-1">b</a> <a href="#cite_ref-helpabout_9-2">c</a> <a href="#cite_ref-helpabout_9-3">d</a> <a href="#cite_ref-helpabout_9-4">e</a> <a href="#cite_ref-helpabout_9-5">f</a> <a href="#cite_ref-helpabout_9-6">g</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://web.archive.org/web/20200317034338/https://help.nla.gov.au/trove/about-the-australian-web-archive">"About the Australian Web Archive"</a>. Trove Help Centre. Archived from <a rel="nofollow" class="external text" href="https://help.nla.gov.au/trove/about-the-australian-web-archive">the original</a> on 17 March 2020. Retrieved 8 May 2020.</cite>

</li>

<li id="cite_note-koerbin2015-10">^ <a href="#cite_ref-koerbin2015_10-0">a</a> <a href="#cite_ref-koerbin2015_10-1">b</a> <a href="#cite_ref-koerbin2015_10-2">c</a> <cite id="CITEREFKoerbin2015" class="citation web cs1">Koerbin, Paul (11 February 2015). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20200501082845/https://www.nla.gov.au/stories/blog/web-archiving/2015/02/11/the-australian-government-web-archive">"The Australian Government Web Archive: Collecting the government's online documentary heritage goes large scale"</a>. National Library of Australia. Archived from <a rel="nofollow" class="external text" href="https://www.nla.gov.au/stories/blog/web-archiving/2015/02/11/the-australian-government-web-archive">the original</a> on 1 May 2020. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-naa-11">^ <a href="#cite_ref-naa_11-0">a</a> <a href="#cite_ref-naa_11-1">b</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.naa.gov.au/information-management/disposing-information/transferring-information/archiving-australian-government-websites">"Archiving Australian Government websites"</a>. National Archives of Australia. Retrieved 8 May 2020.</cite>

</li>

<li id="cite_note-12"><a href="#cite_ref-12">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.nla.gov.au/what-we-collect/archived-websites">"Archived websites"</a>. National Library of Australia. 7 December 2018. Retrieved 6 May 2020.</cite>

</li>

<li id="cite_note-13"><a href="#cite_ref-13">^</a> NOTE: AWA help page says 400 tb, 8 billion records

</li>

<li id="cite_note-phone-14"><a href="#cite_ref-phone_14-0">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.southernphone.com.au/Blog/2019/April/australias-web-archive">"Check Out Australia's Web Archive"</a>. Southern Phone. 11 April 2019. Retrieved 8 May 2020.</cite>

</li>

<li id="cite_note-hometrove-15"><a href="#cite_ref-hometrove_15-0">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://trove.nla.gov.au/website">"Australian Web Archive"</a>. Trove. Retrieved 8 May 2020.</cite>

</li>

<li id="cite_note-16"><a href="#cite_ref-16">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://trove.nla.gov.au/website?q&adv=y">"Australian Web Archive - Advanced Search"</a>. Trove. Retrieved 8 May 2020.</cite>

</li>

<li id="cite_note-National_Library_of_Australia_2020-17"><a href="#cite_ref-National_Library_of_Australia_2020_17-0">^</a> <cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://www.nla.gov.au/what-we-collect/archived-websites">"Archived websites"</a>. National Library of Australia. 23 March 2020. Retrieved 8 May 2020.</cite>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li><a rel="nofollow" class="external text" href="https://trove.nla.gov.au/website">Official website</a></li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-01-23" href="https://en.wikipedia.org/wiki/?title=Australian_Web_Archive&oldid=1271228754">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>